最近，培训预培训方法在以任务为导向的对话框（TOD）系统中表现出了很大的成功。但是，大多数现有的预培训模型用于TOD专注于对话的理解或对话生成，但并非两者兼而有之。在本文中，我们提出了Space-3，这是一种新型的统一的半监督预培训的预训练的对话模型，从大规模对话CORPORA中学习有限的注释，可以有效地对广泛的下游对话任务进行微调。具体而言，Space-3由单个变压器中的四个连续组件组成，以维护TOD系统中的任务流：（i）对话框编码模块编码对话框历史记录，（ii）对话框理解模块以从任一用户中提取语义向量查询或系统响应，（iii）一个对话框策略模块，以生成包含响应高级语义的策略向量，以及（iv）对话框生成模块以产生适当的响应。我们为每个组件设计一个专门的预训练目标。具体而言，我们预先培训对话框编码模块，使用跨度掩码语言建模，以学习上下文化对话框信息。为了捕获“结构化对话框”语义，我们通过额外的对话注释通过新颖的树诱导的半监视对比度学习目标来预先培训对话框理解模块。此外，我们通过将其输出策略向量与响应响应的语义向量之间的L2距离最小化以进行策略优化，从而预先培训对话策略模块。最后，对话框生成模型由语言建模预先训练。结果表明，Space-3在八个下游对话框基准中实现最新性能，包括意图预测，对话框状态跟踪和端到端对话框建模。我们还表明，在低资源设置下，Space-3比现有模型具有更强的射击能力。

具有对比性学习目标的预训练方法在对话了解任务中表现出了显着的成功。但是，当前的对比学习仅将自调查的对话样本视为正样本，并将所有其他对话样本视为负面样本，即使在语义上相关的对话框中，也会强制执行不同的表示。在本文中，我们提出了一个树木结构化的预培训对话模型Space-2，该模型从有限标记的对话框和大规模的无标记的对话框COLPORA通过半监督的对比度预培训来学习对话框表示。具体而言，我们首先定义一个通用的语义树结构（STS），以统一不同对话框数据集的注释模式，以便可以利用所有标记数据中存储的丰富结构信息。然后，我们提出了一个新颖的多视图分数功能，以增加共享类似STS的所有可能对话框的相关性，并且在监督的对比预训练期间仅推开其他完全不同的对话框。为了充分利用未标记的对话，还增加了基本的自我监督对比损失，以完善学习的表示。实验表明，我们的方法可以在DialogLue基准测试中实现新的最新结果，该基准由七个数据集和四个流行的对话框组成。为了获得可重复性，我们在https://github.com/alibabaresearch/damo-convai/tree/main/main/space-2上发布代码和数据。

命名实体识别（NER）任务旨在识别属于人，位置，组织等预定语义类型的文本中的实体。平面实体的最新解决方案NER通常因捕获捕获基础文本中的细粒语义信息。现有的基于跨度的方法克服了这一限制，但是计算时间仍然是一个问题。在这项工作中，我们提出了一个基于跨度的新型NER框架，即全球指针（GP），该框架通过乘法注意机制来利用相对位置。最终目标是实现一个全球观点，以考虑开始和最终位置以预测实体。为此，我们设计了两个模块来识别给定实体的头部和尾部，以使训练和推理过程之间的不一致。此外，我们引入了一种新型的分类损失函数，以解决不平衡标签问题。在参数方面，我们引入了一种简单但有效的近似方法来减少训练参数。我们在各种基准数据集上广泛评估GP。我们的广泛实验表明，GP可以胜过现有的解决方案。此外，实验结果表明，与软马克斯和熵替代方案相比，引入的损失函数的功效。

预先训练的模型已经证明是强大的增强面向任务的对话系统。但是，目前的预训练方法主要关注增强对话的理解和生成任务，同时忽略对话策略的开发。在本文中，我们提出了一个小说预先训练的对话模型，明确地通过半监督学习明确地从有限标记的对话框和大规模未标记的对话框中学习对话策略。具体而言，我们在预训练期间介绍一个对话框预测任务，以便在预训练中进行策略优化，并使用一致性正则化术语在未标记的对话的帮助下优化学习的表示。我们还实施了一个浇注机制来称量合适的未标记对话框样本。经验结果表明，星系大大提高了面向任务为导向的对话系统的性能，并在基准数据集中实现了新的最先进结果：车载，多种多纤2.0和多纺，改善其端到端合并分数2.5,5.3和5.5分。我们还显示Galaxy比各种低资源设置下的现有模型更强大的少量射击能力。

Credit assignment problem of neural networks refers to evaluating the credit of each network component to the final outputs. For an untrained neural network, approaches to tackling it have made great contributions to parameter update and model revolution during the training phase. This problem on trained neural networks receives rare attention, nevertheless, it plays an increasingly important role in neural network patch, specification and verification. Based on Koopman operator theory, this paper presents an alternative perspective of linear dynamics on dealing with the credit assignment problem for trained neural networks. Regarding a neural network as the composition of sub-dynamics series, we utilize step-delay embedding to capture snapshots of each component, characterizing the established mapping as exactly as possible. To circumvent the dimension-difference problem encountered during the embedding, a composition and decomposition of an auxiliary linear layer, termed minimal linear dimension alignment, is carefully designed with rigorous formal guarantee. Afterwards, each component is approximated by a Koopman operator and we derive the Jacobian matrix and its corresponding determinant, similar to backward propagation. Then, we can define a metric with algebraic interpretability for the credit assignment of each network component. Moreover, experiments conducted on typical neural networks demonstrate the effectiveness of the proposed method.

This paper focuses on designing efficient models with low parameters and FLOPs for dense predictions. Even though CNN-based lightweight methods have achieved stunning results after years of research, trading-off model accuracy and constrained resources still need further improvements. This work rethinks the essential unity of efficient Inverted Residual Block in MobileNetv2 and effective Transformer in ViT, inductively abstracting a general concept of Meta-Mobile Block, and we argue that the specific instantiation is very important to model performance though sharing the same framework. Motivated by this phenomenon, we deduce a simple yet efficient modern \textbf{I}nverted \textbf{R}esidual \textbf{M}obile \textbf{B}lock (iRMB) for mobile applications, which absorbs CNN-like efficiency to model short-distance dependency and Transformer-like dynamic modeling capability to learn long-distance interactions. Furthermore, we design a ResNet-like 4-phase \textbf{E}fficient \textbf{MO}del (EMO) based only on a series of iRMBs for dense applications. Massive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, \eg, our EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass \textbf{SoTA} CNN-/Transformer-based models, while trading-off the model accuracy and efficiency well.

Supervised Question Answering systems (QA systems) rely on domain-specific human-labeled data for training. Unsupervised QA systems generate their own question-answer training pairs, typically using secondary knowledge sources to achieve this outcome. Our approach (called PIE-QG) uses Open Information Extraction (OpenIE) to generate synthetic training questions from paraphrased passages and uses the question-answer pairs as training data for a language model for a state-of-the-art QA system based on BERT. Triples in the form of <subject, predicate, object> are extracted from each passage, and questions are formed with subjects (or objects) and predicates while objects (or subjects) are considered as answers. Experimenting on five extractive QA datasets demonstrates that our technique achieves on-par performance with existing state-of-the-art QA systems with the benefit of being trained on an order of magnitude fewer documents and without any recourse to external reference data sources.

Transformer has achieved impressive successes for various computer vision tasks. However, most of existing studies require to pretrain the Transformer backbone on a large-scale labeled dataset (e.g., ImageNet) for achieving satisfactory performance, which is usually unavailable for medical images. Additionally, due to the gap between medical and natural images, the improvement generated by the ImageNet pretrained weights significantly degrades while transferring the weights to medical image processing tasks. In this paper, we propose Bootstrap Own Latent of Transformer (BOLT), a self-supervised learning approach specifically for medical image classification with the Transformer backbone. Our BOLT consists of two networks, namely online and target branches, for self-supervised representation learning. Concretely, the online network is trained to predict the target network representation of the same patch embedding tokens with a different perturbation. To maximally excavate the impact of Transformer from limited medical data, we propose an auxiliary difficulty ranking task. The Transformer is enforced to identify which branch (i.e., online/target) is processing the more difficult perturbed tokens. Overall, the Transformer endeavours itself to distill the transformation-invariant features from the perturbed tokens to simultaneously achieve difficulty measurement and maintain the consistency of self-supervised representations. The proposed BOLT is evaluated on three medical image processing tasks, i.e., skin lesion classification, knee fatigue fracture grading and diabetic retinopathy grading. The experimental results validate the superiority of our BOLT for medical image classification, compared to ImageNet pretrained weights and state-of-the-art self-supervised learning approaches.

Knowledge graph embedding (KGE), which maps entities and relations in a knowledge graph into continuous vector spaces, has achieved great success in predicting missing links in knowledge graphs. However, knowledge graphs often contain incomplete triples that are difficult to inductively infer by KGEs. To address this challenge, we resort to analogical inference and propose a novel and general self-supervised framework AnKGE to enhance KGE models with analogical inference capability. We propose an analogical object retriever that retrieves appropriate analogical objects from entity-level, relation-level, and triple-level. And in AnKGE, we train an analogy function for each level of analogical inference with the original element embedding from a well-trained KGE model as input, which outputs the analogical object embedding. In order to combine inductive inference capability from the original KGE model and analogical inference capability enhanced by AnKGE, we interpolate the analogy score with the base model score and introduce the adaptive weights in the score function for prediction. Through extensive experiments on FB15k-237 and WN18RR datasets, we show that AnKGE achieves competitive results on link prediction task and well performs analogical inference.

Digital engineering transformation is a crucial process for the engineering paradigm shifts in the fourth industrial revolution (4IR), and artificial intelligence (AI) is a critical enabling technology in digital engineering transformation. This article discusses the following research questions: What are the fundamental changes in the 4IR? More specifically, what are the fundamental changes in engineering? What is digital engineering? What are the main uncertainties there? What is trustworthy AI? Why is it important today? What are emerging engineering paradigm shifts in the 4IR? What is the relationship between the data-intensive paradigm and digital engineering transformation? What should we do for digitalization? From investigating the pattern of industrial revolutions, this article argues that ubiquitous machine intelligence (uMI) is the defining power brought by the 4IR. Digitalization is a condition to leverage ubiquitous machine intelligence. Digital engineering transformation towards Industry 4.0 has three essential building blocks: digitalization of engineering, leveraging ubiquitous machine intelligence, and building digital trust and security. The engineering design community at large is facing an excellent opportunity to bring the new capabilities of ubiquitous machine intelligence and trustworthy AI principles, as well as digital trust, together in various engineering systems design to ensure the trustworthiness of systems in Industry 4.0.